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1.  INTRODUCTION  3.  NAVAL  APPLICATIONS 


The  U.S  Navy  has  a  strong  interest  in  conducting  atmo¬ 
spheric  research  in  support  of  operational  mesoscale 
forecasts  used  to  derive  atmospheric  conditions  deemed 
of  strategic  interest  within  the  littoral.  This  effort  will 
require  relatively  fine  horizontal  resolution  (generally  less 
than  10  km)  to  accurately  represent  the  complex  flow 
and  thermodynamic  conditions  which  typically  reside 
near  the  coastal  regions  of  the  globe.  The  task  is  made 
even  more  difficult  by  the  need  to  produce  the  required 
high  resolution  products  within  an  operationally  viable 
time  frame.  It  is  toward  this  end  that  the  Naval  Research 
Laboratory,  in  collaboration  with  the  High  Performance 
Computing  Section  of  the  National  Oceanic  and  Atmo¬ 
spheric  Administration’s  Forecasts  Systems  Laboratory 
(NOAA  FSL),  has  embarked  on  developing  a  parallel 
version  of  the  Coupled  Oceanic  Atmospheric  Mesoscale 
Prediction  System  (COAMPS,  Hodur  1997).  The 
COAMPS  is  a  nonhydrostatic  fully  compressible  finite  ' 
difference  model  which  uses  the  Arakawa-C  grid  stag- 
ger.The  purpose  of  this  paper  is  to  provide  a  brief  over¬ 
view  of  the  effort  to  parallelize  COAMPS  together  with 
two  examples  illustrating  the  naval  need  for  parallel  com¬ 
putational  capability. 

2.  PARALLEL  APPROACH 

The  initial  implementation  of  the  parallel  version  of 
COAMPS  will  employ  the  Scalable  Modeling  System 
(SMS)  developed  at  NOAA’s  FSL  (Hart  et  al.  1994, 
Henderson  et  al.  1994,  Rodriguez  et  al.1994).  The  SMS 
is  a  portable  high-level  library  which  references  either 
internal  native  message  passing  calls  and/or  the  stan¬ 
dard  Message  Passing  Interface  (MPI).  The  SMS  con¬ 
sists  of  the  following  three  components:  (i)  the  Nearest 
Neighbor  Tool,  which  handles  the  domain  decomposi¬ 
tion,  data  exchange,  and  a  interface  to  the  message 
passing  utilities;  (ii)  a  parallel  Pre-processor;  and  (iii)  the 
Scalable  Run-time  System  which  provides  scheduling 
services,  name  services  and  I/O  functions.  The  SMS 
was  established  to  provide  for  an  efficient  means  of  con¬ 
ducting  parallel  10,  and  to  minimize  the  changes  in  the 
initial  sequential  code  through  the  use  of  a  parallelizing 
pre-processor,  and  to  allow  one  source  code  to  execute 
on  several  platforms. 


*  Corresponding  author  address:  Dr.  Jerome  M.  Schmidt, 
Naval  Research  Laboratory,  7  Grace  Hopper  Ave., 
Monterey,  California,  93943;  e-nnail:schmidt@nrlmry.navy.mil. 

^NOAA  Forecast  Systems  Laboratory. 

^  Cooperative  Institute  for  Research  in  the  Atmosphere. 


Applied  and  research  orientated  projects  in  the  littoral 
are  two  typical  examples  of  Naval  applications  requiring 
parallel  computational  capability.  A  prime  example  of  an 
applied  project  surrounds  the  use  of  COAMPS  in  the  On- 
Scene  Tactical  Atmospheric  Forecast  Capability 
(STAFC).  The  objective  of  STAFC  is  to  provide  an  end- 
to-end,  on-scene,  high  resolution  forecast  capability 
using  low-end  computer  hardware.  The  goal  is  to  use 
locally  available  observations  to  improve  the  prediction 
of  the  atmospheric  state  variables  In  the  littoral  and  to 
provide  tactical  weather  parameters  for  ship  self-defense 
(such  as  cloud  parameters,  visibility,  electro-magnetic/ 
electro-optical  propagation  characteristics).  The  primary 
components  of  STAFC  consist  of  COAMPS,  the  Tactical 
Environmental  Data  Subsystem  (TEDS)  database 
server,  a  graphical  user  interface,  www-based  product 
server,  and  other  visualization  tools. 

Fig.  1  shows  the  configuration  of  the  system  and  com¬ 
munications  links  used  on  a  test  of  the  system  aboard 
the  USS  Nimitz  in  June  of  1997.  The  model  required 
nearly  12  hours  to  complete  a  nested  (55x55x20  and 
52x52x20  configuration  using  horizontal  grid  spacing  of 
45  and  15  km,  respectively)  full  physics  36  hour  forecast 
on  an  HP750.  While  this  particular  example  was  run 
onboard  a  ship  (we  believe  this  to  be  the  first  time  in  US 
Naval  history  that  a  mesoscale  weather  forecast  model 
was  run  and  visualized  onboard  an  aircraft  carrier),  a 
more  typical  scenario  will  involve  running  the  model  on  a 
4-8  processor  computer  at  one  of  several  Naval  opera¬ 
tional  centers.  The  overall  goal  of  the  parallel  develop¬ 
ment  will  be  to  insure  the  timely  completion  of  the  high- 
resolution  runs  for  use  in  an  operational  environment 
without  the  need  to  sacrifice  on  model  numerics  and/or 
physics, 

A  second  example  of  parallel  development  centers 
around  research  applications  of  complex  phenomenon 
within  the  littoral.  For  this  example  we  present  observa¬ 
tions  and  modeling  results  from  a  study  of  the  Kaus  flow 
over  the  Arabian  Gulf  (AG).  The  Kaus  is  a  wintertime, 
pre-trough,  moist  southerly  flow  over  the  AG  that  can,  on 
occasion,  reach  gale  force  strength  (Perrone  1979).  His 
depiction  of  the  Kaus  was  one  of  a  cross-isobaric  flow 
along  the  Zagros  mountains  of  western  Iran  (see  his  Fig. 
3.2),The  observations  suggest  that  the  strongest  south¬ 
erly  flow  typically  occurs  over  the  eastern  AG  as  the  flow 
responds  to  “channeling”  along  the  Zagros  and  rapid 
pressure  falls  associated  with  wave-induced  lee  cyclo¬ 
genesis  over  the  northern  AG. 
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Fig.  1  Schematic  diagram  showing  STAFC  configuration  for  the 
USS  Nimitz  test  conducted  In  June  1997.  The  Figure  was  provided 
courtesy  of  John  Cook  of  NRL. 

More  recent  observational,  theoretical,  and  numerical 
studies  have  shown  that  the  windward  slope  of  major 
mountain  chains  is  a  region  conducive  to  the  develop¬ 
ment  of  low-level  along-barrier  jets  or  coastally  trapped 
surges  (Griffiths  and  Hopfinger  1983;  Dorman  1985; 
Pierrehumbert  and  Wyman  1985;  Mass  and  Albright 
1987;  Burk  and  Thompson  1996).  The  dynamics  of  these 
phenomenon  have  been  tied  to  Kelvin  waves,  relatively 
shallow  coastally  trapped  density  currents  (1000m  deep 
or  less),  and  columnar  disturbances  which  have  pertur¬ 
bations  that  tend  to  scale  with  the  depth  of  orography. 
The  relationship  of  the  modeled  Kaus  flow  to  these  basic 
models  of  windward  flow  adjustment  along  a  mesoscale 
barrier  will  be  presented  in  Section  6. 

4.  SYNOPTIC  CONDITIONS  DURING  7-8  FEBRU¬ 
ARY,  1995 

The  time  period  of  interest  to  this  study  covers  the  Kaus 
flow  conditions  that  developed  in  the  pre-trough  environ¬ 
ment  between  0000  7  February  and  0000  8  February 
1995  (Figs.  2  and  3).  The  evolution  of  the  synoptic-scale 
features  closely  follows  the  description  of  other  Kaus 
events  provided  by  Perrone  (1979).  In  this  case  the  syn¬ 
optic-scale  wave  was  accompanied  by  a  strong  subtropi¬ 
cal  jet  of  80  m/s,  cyclogenesis  in  the  northern  gulf  region, 
and  the  development  of  moderate  southerly  flow  near 
the  surface  ahead  of  an  advancing  cold  front  (Fig.  3). 
The  flow  throughout  the  troposphere  in  the  Gulf  region 
was  generally  from  the  west/southwest  in  advance  of  the 
middle-level  trough  except  near  the  surface  where  an 
along-barrier  south-southeasterly  flow  became  estab¬ 
lished  on  the  windward  slope  of  the  Zagaros  (Fig. 3). 
Special  ship  observations  (Fig.  4)  taken  over  the  eastern 
portion  of  the  gulf  between  0700  and  0800  UTC  reveal 
the  along  barrier  flow  took  the  form  of  a  low-level  jet  of 
14-18  m/s  located  within  the  marine  atmospheric  bound¬ 
ary  layer  near  250m  AGL.  The  presence  of  the  jet  is  also 


captured  at  a  similar  level  In  the  surface-based  rawin- 
sonde  released  just  upwind  of  the  AG  (Fig.  5). 


Fig  2.  NOGAPS  500mb  geopotential  height  (dashed  lines)  and 
wind  speed  (shaded  area)  analysis  for  1200  UTC  7,  Feb  1995. 
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Fig.  3  Surface  observation  at  1200  UTC  7  Feb  1995. 

5.  EXPERIMENTAL  DESIGN 

The  model  results  for  this  case  were  generated  using  the 
atmospheric  component  of  the  Naval  Research  Labora¬ 
tory’s  Coupled  Ocean/Atmosphere  Mesoscale  Prediction 
System  (COAMPS)  (see  Hodur  1997  fora  complete 
description  of  COAMPS).  The  simulation  was  initialized 
with  NAVY’S  global  model  analyses  at  0000  UTC  7  Feb¬ 
ruary  1995  with  enhancement  from  the  available  rawin- 
sondes,  pibal,  satellite  winds  and  SSMl  winds  data  and 
run  for  a  24  hour  period  chosen  to  capture  the  evolution 
of  the  pre-frontal  Kaus  flow  over  the  Persian  Gulf.  The 
NOGAPS  12  and  24  hour  forecasts  from  the  initial  time 
were  used  as  the  lateral  boundary  conditions  using  a 
Davies  (1976)  nudging  condition.  For  this  initial  simula¬ 
tion,  moisture  was  treated  as  a  passive  tracer. 

A  fixed  3:1  ratio  was  used  to  define  the  grid  spacing 
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(108,  36,  12  km)  between  the  three  one-way  interacting 
nests  used  in  the  simulations.  A  stretched  vertical  coordi¬ 
nate  was  used  to  vary  the  vertical  resolution  from  20  m 
near  the  surface  to  a  maximum  of  750  m  in  the  lower 
stratosphere.  The  model  top  was  placed  at  22  km.  The 
topography  (see  Fig.  6)  for  ail  experiments  was  obtained 
from  the  Navy’s  1  km  global  data  base.  The  soil  moisture 
and  temperature  profiles  were  derived  from  the  lowest 
atmospheric  analysis  and  the  Navy’s  global  climatologi¬ 
cal  data  base. 
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Fig  4. 1500  UTC  7  Feb.,  1995  radiosonde  released  over  the  Persian 
Gulf  from  the  ship  Silas  Bent.  The  location  of  the  ship  is  marked  as  a 
solid  dot  in  Fig.  4. 


spread  rapidly  northward  in  a  manner  similar  to  the 
descriptions  of  the  coastally  trapped  moist  surges  that 
occur  along  the  west  US  coast  (Mass  and  Albright  1 987). 
The  surge  was  confined  within  a  Rossby  radius  of  defor¬ 
mation  of  the  Zagros  and  maintained  a  steady  propaga¬ 
tion  of  approximately  12  m/s  for  the  entire  simulation 
before  finally  passing  out  the  northern  boundary  of  the 
nest  near  0000  UTC  on  the  8th  (Fig.  7b) 


Fig.  5  Skew-T  at  0000  UTC  8  Feb,  1995.  Sounding  location  is 
given  by  the  +  sign  in  Fig.  6. 


6.  MODEL  RESULTS 

a.  Low-level  regional-scale  flow  and  thermodynamic 
structure 

Plots  of  the  low-level  flow,  thermal  structure,  and  mois¬ 
ture  fields  are  shown  in  Figs.7a,b  for  a  24  hour  period 
covering  the  pre-trough  period  when  a  predominantly 
southerly  Kaus  flow  existed  over  the  Persian  Gulf  region. 
The  evolution  in  the  moisture  field  shows  that  parcels 
residing  or  passing  through  the  Persian  Gulf  region  had 
three  primary  source  regions  consisting  of;  (i)  dry  drain¬ 
age  flow  off  the  Hajar  and  Zagros  Mountains,  (ii)  moist 
marine  air  passing  through  the  straights  of  Hormuz,  and 
(iii)  a  mixture  of  continental  and  marine  air  passing  over 
southern  Saudi  Arabia  along  the  western  flank  of  the 
Hajar  mountains.  Further  moistening  of  this  southerly  air 
stream  occurred  as  a  result  of  evaporation  over  the  rela¬ 
tively  warm  Gulf  waters.  This  resulted  in  low-level  mixing 
ratio  values  over  the  Gulf  of  10  to  15  g/kg  between  0000 
and  1500  UTC  (Figs.  5b-c)  which  corresponds  well  with 
he  values  obtained  from  the  ship  rawinsonde  shown  in 
Fig.  4. 


Fig  6.  Terrain  height  plot  of  COAMPS  grid  3  (contour  interval  is  200 
m).  The  solid  lines  mark  the  cross  section  locations  shown  in  Fig.  7 
and  8.  The  dot  Indicates  the  ship  location  shown  in  Fig.  3.  The  cross 
indicates  the  surface  data  location  shown  in  Fig.  9. 


After  1200  UTC  (Fig.  7a)  the  narrow  moisture  plume 


tThe  model  results  suggest  that  the  moisture  source  over 
the  AG  emanating  from  the  interior  region  of  southern 
Saudi  Arabia  had  its  origin  in  the  inland  penetration  of 
the  marine  layer  along  the  southern  Arabian  coast.  By 
1200  UTC,  the  sea  breeze  circulation  had  penetrated 
inland  approximately  150  km  forming  a  convergence 
zone  over  500  km  in  length.  Further  inland  penetration 
occurred  with  the  onset  of  nocturnal  cooling  as  drainage 
flows  developed  on  the  northern  slope  of  the  Omani 
mountains  and  began  to  advect  the  residual  moisture 
northward  with  time. 

As  the  cooling  intensified,  the  drainage  flow  ultimately 
took  on  a  density  current  structure  with  a  well  defined 
leading  edge  gradient  in  the  thermal,  moisture,  and  wind 
fields  (Figs.  7b  and  Fig  8b).  The  current  was  approxi¬ 
mately  1  km  deep,  propagated  northward  at  approxi¬ 
mately  15  m/s,  and  had  peak  frontal  relative  speeds  of  1 
to  5  m/s  [15-20  m/s  ground  relative  flow].  By  0000  8 
February  (Figs.  7b  and  8b),  the  density  current  had  prop¬ 
agated  northward  into  a  broad  valley  that  extends  south- 
westward  from  Katar  (see  Fig.  6).  Diurnal  heating  over 
the  Interior  desserts  during  the  day  acted  to  mix  this 
moisture  upward  to  a  level  of  approximately  2  km.  This 
process  effectively  altered  the  EM/EO  characteristics 
over  the  AG  as  the  elevated  mixed  layer  propagated 
northward  over  the  water  with  time. 

This  general  sequence  in  the  model  moisture  evolution  o 
is  supported  by  surface  observations  taken  In  the  interior 
of  Saudi  Arabia  west  of  the  Hajar  mountains  which 
shows  a  similar  diurnal  trend  in  the  wind,  temperature, 
and  moisture  fields  over  a  60  hour  cycle  (Fig. 10).  Given 
the  quiescent  nature  of  the  winter-time  flow  In  the  region, 
we  suspect  that  the  sequence  of  events  just  described  is 
rather  common  and  may  thus  play  a  dominate  role  in 
determining  the  wintertime  characteristics  of  the  flow  and 
moisture  fields  over  the  Gulf.  The  role  of  this  flow  in  the 
formation  and  maintenance  of  the  along-barrier  surge 
evident  In  Figs.  7  and  8  is  described  in  the  next  section. 

c.  The  Kaus 

The  modeled  moisture  surge  originated  near  a  region  of 
low-level  convergence  over  the  southern  AG.  This  con¬ 
vergence  zone  formed  between  the  drainage  flow  off  the 
Zagros  mountains  and  the  merger  of  the  two  air  currents 
which  resulted  from  a  split  in  the  low-level  southerly  flow 
about  the  north/south  orientated  Hajar  mountains.  The 
convergence  was  particular  strong  owing  to  the  acceler¬ 
ation  that  occurred  in  both  currents  upwind  from  the  Gulf 
and  the  large  directional  differences  between  the  cur¬ 
rents  as  they  met  in  the  lee  of  the  Hajar  mountains. 

The  large  directional  differences  between  the  two  cur¬ 
rents  resulted,  in  part,  to  the  forcing  of  the  local  orogra¬ 
phy  as  the  flow  was  channeled  toward  the  west  after 
passing  through  the  narrow  Straights  of  Hormuz.  The 


cyclonic/anti-cyclonic  turning  in  the  eastern  and  western 
branches  of  the  split  flow  in  the  lee  of  the  Hajars  is  also 
indicative  of  the  response  one  would  expect  from  low 
Froude  number  (Fr=U/NH)  flow  about  an  isolated  bar- 
rier.Taking  a  low-level  average  of  the  winds  (10  m/s)  and 

stability  (.01  s"'')  over  the  lowest  1000m  and  using  repre¬ 
sentative  terrain  height  of  2000  m  generates  a  local  Fr 
number  estimate  of  approximately  0.5  at  0600  UTC 
which  is  In  the  range  where  we  expect  significant  low- 
level  blocking,  flow  splitting,  and  the  presence  of  lee  vor¬ 
tices  (Smolarkiewicz  and  Rotunno  1989). 

It  is  interesting  that  the  numerical  solutions  of  low  Fr 
number  flow  about  more  isolated  peaks  also  contain  jet 
like  structures  that  Initially  form  along  the  flank  of  the  bar¬ 
rier  and  which  can  extend  well  into  the  lee  (see  Fig.  1  of 
Smolarkiewicz  and  Rotunno  1989).  For  finite  mesoscale 
barriers  such  as  the  Hajars,  these  regions  of  accelera¬ 
tion  about  the  flank  may  become  particularly  strong  as 
the  flow  first  becomes  blocked  along  the  upwind  portion 
of  the  barrier  forming  a  strong  along  barrier  jet  (Fig.  6b). 
Such  a  jet  was  evident  in  the  first  six  hours  of  the  simula¬ 
tion  and  again  after  1800  UTC  as  the  lower  layers  stabi¬ 
lized  due  to  nocturnal  cooling  and  the  passage  of  the 
density  current.  The  presence  of  this  jet  Is  corroborated 
by  the  sounding  data  presented  in  Fig.  5. 

d.  surge  evolution 

After  its  formation,  the  surge  moved  at  a  uniform  speed 
of  approximately  12  m/s  up  the  coast  of  Iran.  This  was 
considerably  slower  than  the  speed  of  the  maximum 
winds  within  the  surge  Indicating  that  there  was  a  system 
relative  speed  toward  the  leading  edge.  Model  derived 
soundings  taken  over  the  central  Gulf  in  the  pre  and 
post-surge  air  masses  (not  shown)  also  reveal  that  the 
wind  and  thermal  perturbations  associated  with  the 
surge  were  confined  vertically  to  within  a  1000  m  of  the 
surface  and  further  indicate  that  the  depth  of  the  cooling 
remained  fairly  constant  well  after  the  leading  edge  had 
past.  Given  these  characteristics,  and  the  fact  that  the 
perturbations  in  the  thermal  and  flow  fields  associated- 
with  the  surge  did  not  scale  well  with  the  overall  height  of 
the  Zagros  (3  km),  suggests  that  the  dynamics  of  the 
feature  relate  more  closely  to  density  current  models 
proposed  by  Mass  and  Albright  (1987)  than  the  columnar 
disturbances  discussed  by  Pierrehumbert  and  Wyman 
(1985). 

The  surge  remained  tied  to  the  low-level  baroclinicity 
associated  with  sloping  MABL  throughout  the  entire  sim¬ 
ulation.  Vertical  cross-sections  taken  normal  to  the  surge 
axis  (Fig.  9)  show  that  a  distinct  jet  core  developed  within 
the  surge  over  the  eastern  Gulf.  This  appears  to  be  a 
characteristic  feature  of  laboratory  generated  gravity  cur¬ 
rents  confined  to  travel  along  a  fixed  boundary  in  a  rotat 
ing  fluid  (Griffiths  and  Hopfinger  1983).  While  diurnal 
forcing  modified  the  slope  of  these  surfaces  somewhat,  a 
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Fig.  7  COAMPS  12  and  24  hour  forecast  of  horizontal  mixing  ratio 
(shaded  area)  and  potential  temperature  (dotted  lines,  every  3  K)  at 
200  meter  height  valid  at  (a)  1200  UTC  7  and  (b)  0000  UTC  8  Feb., 
1995. 

sensitivity  case  run  without  radiation,  reproduced  this 
feature  at  nearly  the  same  magnitude.  This  suggests  that 
the  current  was  driven  primarily  by  the  dynamics  result¬ 
ing  from  the  adjustments  to  gravity  in  a  rotating  fluid. 

7.  Summary 

A  brief  overview  the  approach  used  in  the  parallel  devel¬ 
opment  of  the  NRL  COAMPS  was  presented.  Two  exam¬ 
ples  of  the  naval  need  for  a  parallel  mesoscale  model 
were  also  discussed.  Results  from  a  high-resolution 
model  simulation  initialized  with  real  data  were  pre¬ 
sented  for  a  pre-trough  southerly  Kaus  flow  over  the  Ara¬ 
bian  Gulf  region.  The  purpose  of  the  case  study  was 


Fig.  8  COAMPS  12  and  24  hours  forecast  of  the  horizontal  wind 
speed  (shaded  area)  and  wind  barb  (m/s)  at  200  meter  height  valid 
at  (a)  1200  UTC  7.  and  (b)  0000  UTC  8  Feb.,  1995.  The  wind  barbs 
are  plotted  every  6  grid  points. 

to  document  the  distribution  of  the  low-level  moisture 
fields  over  the  AG  which  have  a  strong  impact  the  EM/ 
EO  propagation  in  the  region.  The  results  indicate  that 
the  low-level  moisture  over  the  Gulf  was  strongly  modi¬ 
fied  by  diurnal  heating/cooling  cycle  that  lead  to  a  moist 
density  current  over  the  interior  regions  of  southern 
Saudi  Araba  and  the  formation  of  a  narrow  coastally 
trapped  surge  that  formed  over  the  eastern  Gulf.  The 
surge  was  initiated  as  low  Froude  number  flow  about  the 
Hajar  mountains  located  upwind  from  the  Gulf  led  to  low- 
level  blocking,  flow  splitting,  and  strong  vorticity  genera¬ 
tion  in  the  lee  of  the  barrier.  The  surge  had  many  charac¬ 
teristics  of  the  surges  that  occur  along  the  west  coast  of 
the  US  including  the  overall  properties  of  a  density 
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curent.  The  surge  also  contained  a  low-level  jet  of  20  m/ 
s  which  formed  over  the  eastern  Gulf.  This  is  consistent 
with  the  descriptions  of  the  stronger  flow  occurring  over 
the  eastern  Gulf  by  Perrone  (1979)  and  Is  also  a  noted 
feature  of  density  currents  confined  to  move  along  a 
fixed  barrier  in  rotating  tank  experiments  Griffiths  and 
Hopfinger  (1983).  The  mesoscale  variability  in  the  low- 
level  thermal,  moisture,  and  flow  fields  over  the  Gulf  that 
resulted  from  these  various  processes  underscores  the 
need  for  high-resolution  prediction  In  the  littoral  in  order 
to  accurately  predict  the  state  variables  and  their  Impact 
on  the  EM/EO  characteristics  over  this  most  vital  region. 


(74.411,  27,4W)  (e4.3W,10241*)  (52.2J<,ftt.SW> 


Fig.  9  Cross  section  A  (position  shown  in  Fig.  4)  analysis  of  wind 
speed  (m/s,  shaded  area)  at  1800  UTC  7  Feb.,  1995. 


level—  170ra 


Fig.  10  Observed  wind  (full  barb  =  5  m/s),  temperature  (C).  and  dew 
point  temperature  (C)  from  the  surface  station  41262  marked  as  a  + 
in  Fig.  4.  Figure  denotes  the  time  in  hours  from  0000  UTC  6  to  1200 
UTC  8  Feb.,  1995  with  3  hourly  time  interval. 
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